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FIELD OF THE INVENTION: 

[0001 ] The invention is related generally to software development systems 
and particularly to a system and method for providing a generic user-interface 
testing framework. 



CROSS-REFERENCE TO RELATED APPLICATIONS: 

[0002] This application is related to copending applications SYSTEM 

AND METHOD FOR PROVIDING A GENERIC USER INTERFACE TESTING 

1 5 FRAMEWORK Application Number , Inventors: Dan Seeman and Zhibin 

Wang, filed March 31, 2004 (Atty. Docket No. BEAS-01512US0); and 
"GENERIC USER INTERFACE TESTING FRAMEWORK WITH LOAD-TIME 

LIBRARIES", Application Number , Inventor: Zhibin Wang and Dan 

Seeman, filed March 30, 2004 (Atty. Docket No. BEAS-01513US0), each of 

20 which applications are incorporated herein by reference. 

BACKGROUND: 

[0003] Modern computing systems often utilize large-scale and/or 
complex software systems. Typical examples of these software systems include 
25 operating systems, application servers, and other complex software applications. 
A key factor in developing and successfully marketing a complex software 
application is maintaining the quality of that application through a process of 
quality control. This typically includes tracking and fixing any inconsistencies, 
faults, or "bugs" that arise in the software code during the development process. 
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[0004] I n most companies the quality control activity is the responsibility 
of a Quality Assurance (QA) team. Software fault analysis is often one of the 
more costly endeavors of the QA cycle of a product, both in terms of human and 
machine resources. Particularly on large and complex software products bugs 
5 can take hours or days to track down, often requiring 30%-40% of the QA teams 
efforts. Any system that can be used to speed up the bug identification and 
tracking process can lead to great improvements in development time and 
produce economic efficiencies for the company. 

[0005] Graphical User Interfaces (GUI's), or User Interfaces (Ul's) pose a 

10 special problem of software testing in that the QA testing of such a Ul must mimic 
the types of interaction a user might make with the interface. Automated U I test 
development systems, suites and tools have been developed, including for 
example the WinRunner test automation system marketed by Mercury Interactive, 
Inc. However, the typical approach of such automated Ul test development tools 

15 requires that the operator have knowledge not just of the test-tool-specific 
scripting language and environment, but also the specific features and idioms of 
the vendor-specific tool environment. The learning curve for these Ul testing tools 
can be significant, and is often of use solely with that one tool. (It is not 
uncommon for new users to spend 6 months or more to "ramp up" on a given 

20 tool, i.e. learn how to install it, use it, and write code to run it so as to produce 
workable, sustainable, reusable Ul automation code). If another type of tool is to 
be used, the tester must again deal with the learning curve of the new tool. 
Furthermore, with the typical test development tool one needs to install the 
vendor specific software to use it, incurring both capital outlay and (local) 

25 resource costs. This initial capital outlay (i.e. the cost of the tool) typically cannot 
be recovered if the original tool is subsequently replaced with a new tool. 
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SUMMARY: 

[0006] A system is herein described for providing a generic Ul testing 
framework. The test framework, together with the systems and methods 
embodying the technology, maps the native test development language and 
5 environment into arbitrary languages and environments. This generic Ul test 
framework insulates test developers from learning the tool-specific scripting 
language and environment. 

[0007] In accordance with one embodiment, the Ul test framework 
provides a set of function interfaces and implementations that cover all generic 
10 Ul testing operations. New users need only map their testing logic to the 
supported library interface in order to use the test tool, or another test tool, 
without having to learn the details of the underlying test-tool-specific scripting 
language. 

[0008] In accordance with another embodiment, the system includes a 
15 mechanism such that test developers can configure which libraries are 
applicable to the desired application(s) to be qualified and the test framework 
will only load those libraries at execution time. With the ability to dynamically load 
only the desired libraries one is able to "snap in" new testing modules as needed 
thus being able to accommodate new product qualification efforts with greater 
20 flexibility and efficiency. Test scripts can bedeveloped into different functional 
modules. By developing modular libraries, we can develop a Ul test framework 
to identify, load and use those libraries that are applicable to the applications 
under test. 

[0009] In accordance with another embodiment, the system includes a 
25 rules based automation script generating engine that removes the necessity of 
test developers from learning the abstract environment, parameters and 
directives used to implement the Ul test automation system. A user interface or 
wizard guides test developers through the various development stages. Upon 
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completion, the test developer is left with a fully functional, end-to-end Ul testing 
automation script. 

BRIEF DESCRIPTION OF THE FIGURES: 
5 [0010] Figure 1 shows a logical diagram of a generic user-interface 
testing framework. 

[0011] Figure 2 shows a flowchart for using a generic user-interface 
testing framework. 

[001 2] Figure 3 shows a logical diagram of dynamic load mechanism for 
1 0 use with a generic user-interface testing framework. 

[001 3] Figure 4 shows a flowchart for using a dynamic load mechanism 
with a generic user-interface testing framework. 

[0014] Figure 5 shows the relationship between a wizard (or other 
editors) and the directives of test logic. 

15 

DETAILED DESCRIPTION: 

[001 5] Graphical User Interfaces (GUI's), or User Interfaces (Ul's) pose a 
special problem of software testing in that the QA testing of such a U I must mimic 
the types of interaction a user might make with the interface. Automated Ul test 

20 development systems, suites and tools have been developed, including for 
example the WinRunner test automation system marketed by Mercury Interactive, 
Inc. Building Ul test cases traditionally involves using the Ul test scripting 
language to construct the test logic of each test case. However, these Ul test 
scripting languages are tied to specific test tools, and are traditionally difficult to 

25 learn and to use. 

[0016] The inventors have reasoned that Ul testing operations can be 
categorized into several dozens of common ones. Those Ul testing operations, 
independent from test tools, are high-level logical abstractions of the actions that 
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a user can perform on a particular Ul. Ul test cases (containing "directives") can 
then be decoupled from the native scripting languages in such a way that Ul test 
cases can be built using the descriptions of those common test operations. A 
Ul test framework can supply the service of an interpretive engine that interprets 
5 the logic in the Ul test cases and function libraries developed to support those 
generic testing operations in test-tool-specific scripting languages. In this way, 
Ul test developers only need to write their automated Ul tests in a 
tool-independent (or indeed a completely tool-absent)format ,such as in an XML, 
spreadsheet or a text form, to capture the test logic. In the test execution 

1 0 process, the Ul framework's interpretive engine reads the directives of the test 
logic to drive tests using the underlying function libraries. 
[0017] The directives of test logic can be captured in any format as long 
as the interpretive engine of the Ul test framework is able to read the format 
correctly and to retrieve the information of the test logic. The directives can be 

15 created or edited using the appropriate editing tool for the chosen format. For 
example, if the directives are captured in the XML format then a text editor, an 
XML editor, or a wizard can be used to create or edit the file containing the 
directives; if the directives are captured in a spreadsheet, then a Microsoft Excel 
spreadsheet tool, or a similar spreadsheet program, can be used. 

20 [0018] Because there are several dozens of well-defined logical Ul testing 
operations, in accordance with one embodiment the Ul test framework can 
provide a utility wizard that displays supported functions and guides test 
developers to pick and choose which functions they would like to use as the 
directives. The interpretive engine of the Ul framework maps the directives into 

25 function libraries that are implemented in the test-tool-specific scripting language. 
[001 9] If, at some later date, the company or the test developers decide 
to use another test tool, then the test cases can be preserved since they only 
contain tool-independent test logic. Test framework developers can implement 
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the same low-level functions in the new scripting language and the interpretive 
engine, thereby preserving and thus building on the value of all the work and 
experience that has been accomplished with the previous tool. Test developers 
familiar with the tool-independent environment are insulated from any change in 
5 the underlying tool. There is no need for them to even know the low level 
mechanism has been switched. 

[0020] In accordance with other embodiments, kits and documentation 
may be provided forthe Ul test framework. New users only need to install the kit 
and follow the documentation to be able to utilize the advancement. 

1 0 [0021 ] The test tool is only required fortest execution. It is not required to 
build test cases, which contains tool-independent test logic. Users can choose 
whether or not to install "online" a core, tool-specific software. If they decide not 
to install the tool-specific software, users can create test cases "offline" at 
another machine, and then simply copy their tool-independent test cases to a 

15 central location where it will then be executed. This also saves cost on buying 
multiple instances of the vendor specific test tool. 

Generic User Interface Test Framework 

[0022] Figure 1 shows a logical overview of a generic user-interface 
20 testing framework 2 in accordance with an embodiment of the invention. Test 
cases 4 contain the directives of tool-independent test logic and are written in a 
predefined format. The directives of test logic can be captured in any format as 
long as the interpretive engine of the Ul test framework is developed and/or 
configured to read the format correctly and to retrieve the information of the test 
25 logic. The directives can be created or edited using the appropriate editing tool 
for the chosen format. For example, if the directives are captured in the XML 
format, a text editor, an XML editor, or a wizard can be used to create or edit the 
file containing the directives; if the directives are captured in the spreadsheet, 
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then an Excel spreadsheet or similar tool can be used. 
[0023] In accordance with one embodiment, an interpretive engine 6 
parses the directives properly, and maps the directives to the underlying 
functions (in one embodiment through the use of libraries 8 or dynamic load 
5 libraries described in further detail below) to execute the Ul action in the 
test-tool-specific environment 10. Both the interpretive engine and the supported 
function libraries are written in the tool-specific scripting language, for example 
WinRunner or any other tool language. 

[0024] Developing a Ul test case includes building directives that capture 
1 0 the logical testflow. A Ul test developer is not required to know test-tool-specific 
scripting language or environment to build test cases. The test developer only 
needs to know how to capture the test logic using well-defined common testing 
operations (such as button-click, link-click, etc). An editing tool may be used to 
help create and/or edit the directives. 
1 5 [0025] If a new tool is used to replace the existing tool, all of the directives 
4 of test logic can be retained. Only the interpretive engine and the function 
libraries need to be re-implemented in the new scripting language. For 
example, switching from WinRunner as the testing tool, to a different testing tool, 
requires only changes to the interpretive engine and/or the function libraries. The 
20 directives themselves, which can be saved as test files, spreadsheets, or some 
other format, may be re-used. 

[0026] Figure 2 shows a flowchart for a process of providing a generic 
user interface testing framework in accordance with an embodiment of the 
invention. As shown in Figure 2, in step 12 the user interface tester/developer 
25 develops user interface testing scripts containing test directives. The scripts may 
be developed either online or offline. In an online environment, the 
scripts/directives can be fed directly to the test environment. In an offline 
environment, the scripts/directives can be saved and processed at a latertime. 
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In step 14, the testing script or directives are communicated to the interpretive 
engine. In step 1 6, the interpretive engine translates the script and/or directives 
and maps the generic commands therein into test suite-specific or test 
tool-specific language. In step 18, the test suite-specific language is used to 
5 drive the test suite in testing the build of the user interface. 

[0027] Within any software corporation, most of the work involved in Ul testing 
lies in building the test cases. Test cases can grow to thousands of lines of code 
(or more), while the number of actual common testing operations is limited. The 
amount of work to continuously build test cases far exceeds that of building an 
1 0 interpretive engine and the supported function libraries. In accordance with the 
present invention, the ability to preserve test cases during the test-tool switch 
provides exceptional value to reduce the cost of test development and 
maintenance. 

15 Dynamic Load Mechanism 

[0028] User interface applications can be distinctly different. For 
example, Internet browser based applications are visually and functionally 
differentfrom the Java (swing/awt)UI applications. To conserve money, time and 
resources, companies often select one Ul automation tool that can accommodate 

20 a variety of application types. This allows companies to share at least 
knowledge and training and in rare cases, tool-specific source code. 
[0029] However, the aggregation of test-tool-specific source code for 
heterogeneous application types creates significant management overhead. The 
management overhead is often so onerous that companies often separate 

25 development efforts in total, resulting in duplicated staffing, efforts and source 
code. Reducing this duplication represents a significant savings for the 
company, not just in project management overhead, staffing, etc. but also in 
source code maintenance, flexibility and scalability. 
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[0030] An embodiment of the present invention addresses this issue by 
including within the generic Ul test framework a mechanism such that test 
developers can configure which libraries are applicable to the desired 
application(s) to be qualified. The test framework will only load those desired 
5 libraries at execution time. With the ability to dynamically load only the desired 
libraries one is able to "snap in" newtesting modules as needed, thus being able 
to accommodate new product qualification efforts with greater flexibility and 
efficiency. 

[0031 ] The core of the dynamic load feature is to implement a well-defined 
1 0 contract system for executing loadable libraries. The higher layer system is able 
to load and execute contract compliant libraries. This system is similar to other 
dynamically loadable library systems. However the present system can be used 
to apply this logic to specific Ul test tool products such that they can be used to 
qualify a wide variety of applications. 
1 5 [0032] A key feature of the system is its flexibility. One is able to use the 
system to test a variety of applications. Moreover the system is generalized 
enough that one is not strictly confined to the implementation language of all the 
loadable libraries. As long as the loadable library entry point is contract 
compliant (and language specific), then subsequent libraries can be written in 
20 any language supported by the test-tool development language. For example, 
most U I test tool languages have an execution interface. One is perfectly free to 
use the execution interface to gain access to and use of any other development 
language. 

[0033] Figure 3 shows a design for a system that includes a dynamic load 
25 library mechanism in accordance with an embodiment of the invention. As shown 
in Figure 3, the system includes a generic testing framework 20 largely as 
described above, including an interpretive engine 24 that receives directives and 
translates them into Ul test-tool-specific language 10. Functions are logically 
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grouped into libraries (26-30), such as Web Functions 26, Swing Functions 28, 
Log Functions 30, etc. Additional libraries may be added as needed. The test 
framework can be configured to identify which libraries are to be loaded. Once 
it is configured, the test framework will only load those specified libraries at run 
5 time, which provide support for properly executing the tests. I n this way, different 
U I tests will only load and use those libraries applicable to the tests. As new tool 
environments are used within a corporation, additional libraries can be included 
in the testing framework to provide support for those environments. 
[0034] Figure 4 shows a flowchart for a process of using a dynamic load 

1 0 mechanism with a generic user interface testing framework in accordance with 
an embodiment of the invention. As shown in Figure 4, in step 32, the test 
framework is configured to identify which run-time load libraries should be 
loaded. In step 34, at run-time the specified load libraries are loaded. In step 
36, the testing framework receives the scripts/directives, typically from a wizard, 

1 5 spreadsheet or other interface as described above. In step 38, the testing script 
or directives are communicated to the interpretive engine. In step 40, the 
interpretive engine translates the script and/or directives and, using the 
dynamically loaded libraries, maps the generic commands therein into test 
suite-specific commands for communications to the test suite or test tool. 

20 

User Interface Automation Wizard 

[0035] After the developer has been abstracted away from writing 
test-tool-specific Ul automation source code the problem of having to learn a 
test-driving language remains. A generic framework such as that described 
25 above is useful in mapping the test-tool-specific language into something more 
familiar to the target (test developing) audience. But even in the best 
circumstances the test developer must learn a new set of directives. The set of 
directives can be abstract and confusing. Furthermore, this knowledge is not 
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transferable, as it is specific to the implementation environment. This task is 
equivalent to that of learning one company's internal (company, project or even 
group specific) Application Programming Interface (API) set. The language itself 
(XML for example) may be common, but the language formats, rules, commands 
5 and directives are typically unique. 

[0036] An embodiment of the invention addresses this issue by including 
a rules- based automation script generating engine that removes the necessity 
of test developers from learning the abstract environment, parameters and 
directives used to implement the Ul test automation system. 

1 0 [0037] I n accordance with an embodiment, a user interface or wizard guides test 
developers through the various development stages. Upon completion of all 
stages, the test developer is left with a fully functional, end-to-end Ul testing 
automation script. This is not a replacement for a Ul automation test-tool-specific 
action recorder. The source code produced by this engine is test tool "agnostic" 

15 and can be used (potentially) with any automation test tool. As long as the 
appropriate framework and libraries have been implemented in the 
test-tool-specific language, the source code produced by this rules engine can 
be used to drive it. 

[0038] Ul test scripting languages are difficult to learn and to use. 

20 Moreover, it is often easy to fall into the "use-once-and-trash-it" trap of Ul 
automation tools. It is particularly easy to fall into this trap when using automation 
test-tool-specific action recorders. The test-tool-specific code produced by 
these action recorders is often relevant only to the specific parameters and 
environment(s) where the recorder was employed. This renders the generated 

25 code useless when it is copied to another location. This locks the developers 
into a perpetual state of writing and re-writing test automation code for similar or 
completely equivalent test scenarios. Companies hemorrhage significant capital 
and resources because of this perpetual cycle. 

Attorney Docket No.: BEAS-01514US0 Express Mail Label No.; EV 385 256 515 US 

kfk/beas/1 51 4us0/1 51 4 usO. application.wpd 



11 



[0039] The solution to this problem is to first map the test-tool-specific 
language into one that is more familiar to the test developing staff, which is 
provided by te generic testing framework described above. A second layer to 
this solution is to provide a rules-based wizard that guides test developers 
5 through the code development sequence. The product of this engine is a 
standardized automation test that can be used with (potentially) any vendor 
specific Ul automation tool. Implementation of this engine/wizard provides 
significant capital savings, day-to-day efficiency, learning curve reduction (even 
elimination) as well as test automation source code maintenance and flexibility. 

1 0 [0040] Figure 5 shows the relationship between the wizard 44 (and other 
editors 46, 48) and the directives 50 of test logic. A rules-based wizard can be 
used to create or edit the document containing the directives. In accordance with 
an embodiment the wizard provides a set of Ul testing operations. The wizard 
can then guide test developers to pick and choose any of the well-defined Ul 

15 testing operations so as to build test cases. These test cases are then 
communicated to the testing framework as directives, and are processed as 
described in detail above. 

[0041] The present invention may be conveniently implemented using a 
20 conventional general purpose or a specialized digital computer or 
microprocessor programmed according to the teachings of the present 
disclosure. Appropriate software coding can readily be prepared by skilled 
programmers based on the teachings of the present disclosure, as will be 
apparent to those skilled in the software art. 
25 [0042] In some embodiments, the present invention includes a computer 
program product which is a storage medium (media) having instructions stored 
thereon/in which can be used to program a computer to perform any of the 
processes of the present invention. The storage medium can include, but is not 
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limited to, any type of disk including floppy disks, optical discs, DVD, CD-ROMs, 
microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, 
DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems 
(including molecular memory ICs), or any type of media or device suitable for 
5 storing instructions and/or data. 

[0043] The foregoing description of the present invention has been 
provided for the purposes of illustration and description. It is not intended to be 
exhaustive or to limit the invention to the precise forms disclosed. Many 
modifications and variations will be apparent to the practitioner skilled in the art. 

1 0 Particularly, while the embodiment of the system is described in the context of a 
WebLogic system, and in combination or use with test development systems 
such as WinRunner, it will be evident that the framework provided may be used 
with other types of applications and systems, including other types of application 
servers, and with other types of test development systems. The embodiments 

15 were chosen and described in order to best explain the principles of the invention 
and its practical application, thereby enabling others skilled in the art to 
understand the invention for various embodiments and with various modifications 
that are suited to the particular use contemplated. It is intended that the scope of 
the invention be defined by the following claims and their equivalence. 

20 
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